This invention relates to methods and apparatus for image processing and more particularly to image enhancement. In particular, the invention relates to methods and apparatus for the enhancement of both video images of natural scenes that contain a wide range of spatial frequencies and of real-world views of natural scenes.
Traditional image enhancement methods suffer from a number of drawbacks. Many traditional image enhancement methods can not effectively enhance images over a wide band of spatial frequencies. For example, one technique enhances an image by changing its spatial frequency content through manipulation of the coefficients of a discrete cosine transform (xe2x80x9cDCTxe2x80x9d) of the image. The method segments the image into 8xc3x978 pixel sections and obtains the cosine transform of each section. This technique fails to capture low frequency components that arise as a result of features that have significant variations in luminance mainly over an area larger than an 8xc3x978 pixel section. Other techniques process an image in the spatial domain. Such techniques typically enhance the image only over a narrow range of frequencies. See E. Peli, xe2x80x9cLimitations Of Image Enhancement For The Visually Impaired,xe2x80x9d Optometry and Vision Science, vol. 6, pp. 15xe2x80x324 (1992); and E. Peli, E. Lee, C. L. Trempe, S. Buzney, xe2x80x9cImage Enhancement For The Visually Impaired: The Effects Of Enhancement On Face Recognitionxe2x80x9d, Journal of Optical Society of America, vol. 11 pp. 1929-1939 (1994).
Therefore, such traditional image enhancement techniques are not suitable for enhancing the images of many natural scenes that contain a wide range of spatial frequencies. Further, human observers detect moving objects that contain a wide band of frequencies more readily than those with a narrow band of frequencies. Thus, the traditional techniques are not appropriate in systems for assisting detection of moving objects, or in systems that provide real-time viewing enhancement of natural scenes.
Traditional methods also can not readily enhance an image while the size of the image changes. For example, the viewer of a digital television display could desire to follow the image of an object that undergoes a large change in its size while maintaining a selected degree of enhancement. The ability to enhance a wide range of frequencies is crucial in such applications. Traditional techniques, such as a DCT method or other band-limited methods are not appropriate for such applications because they provide a limited range of spatial frequencies of the image.
In addition, traditional enhancement methods, both in the spatial domain and in the frequency domain, typically manipulate a large fraction of pixels. As a result, their use in the enhancement of color pictures requires tracking the color content of many pixels while the computation changes the luminance of those pixels.
Accordingly, it is an object of this invention to provide methods and apparatus for enhancing images over a wide band of spatial frequencies.
It is another object of the invention to provide methods and apparatus that can readily enhance such images over a reasonable range of image sizes.
It is yet another object of the invention to provide methods and apparatus for real-time viewing enhancement of natural scenes.
It is a further object of the invention to provide methods and apparatus for better enhancement of color pictures.
It is yet a further object of the invention to provide methods and apparatus for expanding field of view of a patient suffering from peripheral field loss.
The invention is next described in connection with illustrated embodiments. It will, however, be obvious to those skilled in the art that various modifications can be made in the embodiments without departing from the spirit or scope of this invention.
The methods and apparatus according to this invention modify an image by 1) locating certain features of the image, such as the boundaries of objects in the image, 2) manipulating such located features to obtain modified features, and 3) adding the modified features to the original image. In particular, one embodiment of the invention employs a two-dimensional Hilbert transform of the image data to create a two-dimensional function, a so-called energy function, whose local maxima correspond to points lying on the boundaries between regions of marked difference in luminance, i.e., edges, or to points corresponding to peaks or troughs in luminance, i.e., bars. The invention further provides techniques to interconnect these maxima, thus delineating the desired features.
An application of this invention is to improve the visibility of video images for people with visual impairment, e.g., cataracts or macular degeneration. In particular, one embodiment of the present invention allows real-time image processing and enhancement of the real-world view for the visually impaired. This embodiment includes a dedicated microprocessor, programmed to extract the boundaries of objects in the field of view, according to the methods of the invention from data inputted from a digital camera. This embodiment also incorporates video equipment to project extracted features onto screens. These screens can be integrated in a wearable real-time image enhancement apparatus, such as a head mounted display (xe2x80x9cHMDxe2x80x9d) display unit.
Another application enhances the real-world view, under reduced visibility conditions such as fog, by projecting the enhanced features, obtained from non-visual sensors, e.g., infrared or radar, on heads-up displays (HUD) of an airplane or of a car windshield. Another application of this invention is to improve the visibility of television images for individuals with visual impairment. Yet other applications relate to the enhancement of satellite and reconnaissance pictures or other military imaging devices, and to the delineation of features of interest in such pictures.
The invention is typically practiced on a digital image that consists of a discrete two-dimensional map of luminance. Some embodiments of the invention represent such images by two dimensional matrices. The invention employs an extension of well known methods for calculating the Hilbert transform of a function in one dimension to obtain a discrete two-dimensional Hilbert transform of a function of the image data.
It is well understood that the one-dimensional Hilbert transform of a function of a single variable can be calculated by 1) obtaining the Fourier transform of the function, 2) obtaining a modified transform function whose values are zero at points where its independent variable is less than zero, and whose values are those of the Fourier transform at points where its independent variable is larger than zero. A third step is to obtain the inverse transform of this modified transform function.
One preferred embodiment of the invention obtains the two-dimensional Hilbert transform of the image data by 1) computing the two-dimensional Fourier transform of the image, 2) obtaining a new two-dimensional transform function whose values in a selected arbitrary contiguous half of the two-dimensional Fourier plane are zero, and whose values correspond to those of the two-dimensional Fourier transform of the image in the other half, and 3) obtaining the inverse Fourier transform of the modified transform function. The real part of the complex inverse Fourier transform of the modified transform function corresponds to the original image and the imaginary part corresponds to the Hilbert transform of the image.
One preferred embodiment of the invention combines the image data with the Hilbert transform of the image data to obtain a new two-dimensional function, a so-called energy function. In particular, the procedure for forming the energy function calls for obtaining the square root of the Pythagorean sum of the image data and of the values of the Hilbert transform at each point, e.g., at each pixel of a digital image.
One embodiment of the invention utilizes the positions of the peaks of the energy function to locate the visually relevant luminance features of the image. It is understood that such peaks correspond to peaks or troughs in luminance, or to those locations in the original image where changes in image intensity profile occur because of the existence of maximal phase congruency among the various Fourier components of the image.
The local maxima of the energy function correspond to points of both minimum and of maximum intensity in the original image data, and also to the boundaries between regions of low and of high luminance. It is not reasonably feasible to classify the maxima of the energy function with respect to the polarity of the corresponding points in the image data based purely on the energy function. Thus, some embodiments of the invention implement a further examination of the image data at each point that corresponds to a maximum of the energy function to label the polarity of each such maximum.
One aspect of the present invention relates to the creation of a map of dots corresponding to the points designated as the maxima of the energy function. The invention optionally employs methods known in the art to connect these dots to produce lines corresponding to the desired features. In addition, the invention provides the capability of manipulating these lines by widening them through convolution with an appropriate windowing function, e.g., a Gaussian with a selected width, or manipulating their intensities, to improve the contrast of the image.
Some embodiments of the invention employ only one arbitrarily selected polarity, i.e., either dark or bright, to display the dots or the contour lines at edges, whereas other embodiments utilize two polarities. A bipolar representation displays an edge with two dots, one dark and the other bright, next to each other. Some embodiments that utilize a bipolar representation examine the unmodified image to select a choice for juxtaposition of the dark and bright dots that corresponds to the sense of the transition of luminance at the corresponding location of the image. Both embodiments represent the polarity of bars in accordance with the polarity in the original image. Other embodiments of the invention use only a single polarity of dots, i.e., light or dark, to represent all bars or edges.
A preferred embodiment of the invention superimposes these modified contour lines onto the original image to obtain a new image in which certain features have been modified, e.g., the boundaries of the objects in the image have been enhanced.
The invention can also enhance color images. Because the invention manipulates only a limited number of pixels, i.e., those corresponding to the visually relevant features of the image, only a few pixels change color due to the enhancement. Thus, the methods of the invention are better in preserving the color of an image than other enhancing techniques.
An alternative method for locating luminance features, such as edges and/or bars, of an image and their polarities employs a plurality of oriented filters to extract the luminance features. The application of a number of filters having different center frequencies, and optionally different bandwidths, to the image provides a plurality of filtered images. One preferred practice of this aspect of the invention assigns to each pixel of a filtered image either a dark, a light, or a gray scale in a manner described below, to provide an assigned image. A contour constructor, such as a programmable digital processor, receives the assigned images, and obtains the features of the original image by application of a set of pre-programmed steps to the assigned images, as described further below.
In another aspect, the invention expands the field of view of patients suffering from loss in their peripheral vision. In particular, one practice of this aspect of the invention obtains a spatially minified image of the natural scene, and extracts the luminance features of the minified image. Video equipment projects the extracted features on a transparent screen disposed before at least one eye of the patient. The patient can readily locate objects in the minified image, and view the located objects directly by scanning her eyes and/or by moving her head.
Another aspect of the invention relates to providing a night vision device that facilitates night mobility of patients who suffer from a loss of night vision. In particular, one practice of the invention obtains an infrared image of a natural scene, extracts the contours of objects in the image according to the teachings of the present invention, and projects the contours onto a see-through visual screen disposed in front of at least one of the patient""s eyes.
One further practice of the invention relates to intraocular implantation of a telescope that is configured as a minifier into a patient""s eye, to provide a wide-angle minified view of a natural scene. Such a minified view helps the mobility of a patient suffering from peripheral field loss.
Thus, the invention attains the objectives set forth above by extracting visually relevant features of an image, manipulating these features to obtain modified features, and superimposing such modified features onto the original image to obtain a modified image.